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1. Introduction 

The 9-Neumann problem is a natural example of a boundary-value problem 
with an elliptic operator but with non-coercive boundary conditions. It is also 
a prototype (in the case of finite-type domains) of a subelliptic boundary-value 
problem, in much the same way that the Dirichlet problem is the archetypal elliptic 
boundary-value problem. In this survey, we discuss global regularity of the d- 
Neumann problem in the L^-Sobolev spaces W'(O) for all non- negative s and also 
in the space C°°(ri). For estimates in other function spaces, suc h as Holder spaces 
and LP-Sobolev spaces, see Q H, ||, || 1^, |l|, || ^ |8|, |6|, pTTl [111, 1123 



128 



145 



122 



109 



143; for questions of real analytic regularity, see, for example, [5l|, |5g|, |7"C 
146, 147, 148| and section 10 of Christ's article in these proceedings. 
We also discuss the closely related question of global regularity of the Bergman 
projection operator. This question is intimately connected with the boundary reg- 
ularity of holomorphic mappings (see, for example, ||ll|, |2^, |l^, |l^, |l^, p6[|). 
For an overview of techniques of partial differential equations in complex analysis, 
1 



see 



jarapM 103, 1131 



2. The existence theory 

Throughout the paper, Q. denotes a bounded domain in C", where n > 1. We 
say that Q, has class boundary \i Q, — {z : p{z) < 0}, where p is a k times 
continuously differentiable real-valued function in a neighborhood of the closure fl 
whose gradient is normalized to length 1 on the boundary bil. We denote the 
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standard L^-Sobolev space of order s by W^Q) (see, for example, |, ^2^ pl9|| ). 
The space of (0, q) forms with coefficients in W'^{fl) is written W^^ ^^"^ norm 

being defined by 

(1) II '^j'^^All = 

where dzj means dzj-^^ A dzj^ A • • • A dzj^ , and the prime indicates that the sum 
is taken over strictly increasing g-tuples J. We will consider the coefficients aj, 
originally defined only for increasing multi-indices J, to be defined for other J 
so as to be antisymmetric functions of the indices. For economy of notation, we 
restrict attention to (0, q) forms; modifications for (p, q) forms are simple (because 
the d operator does not see the dz differentials). 
The d operator acts as usual on a (0, q) form via 



(2) 9[Y:^^d-zj)=YT:'^'^-^: 



ZjJ- 



The domain of d : L'^^ ^-^{fl) — > L^(, ^^-^^ (fi) consists of those forms u for which 
du, defined in the sense of distributions, belongs to L^^ ^^j^j (fi). ft is routine to 
check that 9 is a closed, densely defined operator from L'^^ ^^{fl) to L'^^ . 

Consequently, the Hilbert-space adjoint d also exists and defines a closed, densely 
defined operator from L^p g^i-^i^) to Lj^ 

Suppose u = Y^'j uj dzj is continuously differentiable on the closure Q, and ip is a 
smooth test form. If the boundary bfl is sufficiently smooth, then pairing u with d^p 
and integrating by parts gives 

(3) {u,di')= ^^'^^K^A + J2 I ^KY'^kK-^da. 

V K ^^f" ^ K -Ibn ClZk 

The same calculation with a compactly supported -0 shows (without any boundary 
smoothness hypothesis) that if u is a square-integrable form in the domain of d , 
then d u — liu, where the formal adjoint is given by the equation 

(4) ^--EE'^'^^-- 

k=l K 

It follows that a continuously differentiable form u is in the domain of d if and 
only if 



fc=i 



~ for every K. 

bn 



The method of Friedrichs moUifiers shows that forms which are continuously 
differentiable on the closure are dense in the intersection of the domains of d 
and d with respect to the graph norm (||u|p + + \\d u|p)^/^ when the 

boundary bQ is sufficiently smooth (see, for instance, §1.2 and Prop. 2.1.1]). 
Also, forms that are continuously differentiable on the closure are dense in the 
domain of d with respect to the graph norm {\\u\\^ + ||9u|p)^/^. 

The fundamental existence theorem for the 9-Neumann problem is due to 
Hormander |B3[. One version of the result is the following. 
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Theorem 1. Let CI be a bounded pseudoconvex domain in C", where n > 2. Let 
D denote the diameter of 57, and suppose 1 < q < n. 

1. The complex Laplacian □ — dd +d d is an unbounded, self-adjoint, surjective 
operator from L^q gj{^) to itself having a bounded inverse Nq (the d-Neumann 
operator). 

2. For all u in q)i^)' ™^ have the estimates 

'D^e' 



(6) \\d Nqu\\ < [ — : j 11^.11 



1/2 



\\dNgu\\ < ^ 

3. // / is a d-closed (0, q) form, then the canonical solution of the equation 
du = f (the solution orthogonal to the kernel of d) is given by u — d Nqf ; 
if f is a d -closed (0, q) form, then the canonical solution of the equation 
d u — f (the solution orthogonal to the kernel of d ) is given by u = dNqf. 

The Hilbert space method for proving Theorem |^ is based on estimating the 
norm of a form u in terms of the norms of du and d u. Hormander discov- 
ered that it is advantageous to introduce weighted spaces L^(ri,e~'^), even for 
studying the unweighted problem. We denote the norm in the weighted space by 
\\u\\ip = llue^"^/^!! and the adjoint of d with respect to the weighted inner prod- 
uct by ■) = e^ d ( • e~^). More generally, one can choose different exponential 

weights in <j-i)' ^(o <?)' ^^'^ ^fo 9-1-1)' 10 ^^^^ method and applications. 

The following identity is the basic starting point. The proof involves integrating 
by parts and manipulating the boundary integrals with the aid of the boundary 
condition for membership in the domain of d . The idea of introducing a 
second auxiliary function a originated with Ohsawa and Takegoshi |131 , 13C | in their 
work on extending square-integrable holomorphic functions from submanifolds. The 
formulation given below comes from the recent work of Siu |141| and McNeal |125|. 
In these papers (see also the freedom to manipulate both the weight factor ip 

and the twisting factor a is essential. 

Proposition 2. Let fl be a bounded domain in C" with class boundary; letu be 
a (0, q) form (where 1 < q < n) that is in the domain of d and that is continuously 
differentiable on the closure fl; and let a and ip be real functions that are twice 
continuously differentiable on Q, with a > 0. Then 

(7) ||Va9-u||^ -I- = V V / a UjKUkKe^"^ da 



EE 



a 



du. 



e-f dV + 2^(y^Y. 1^ dzK.%u 

V K j=l ' 



4 



HAROLD P. BOAS AND EMIL J. STRAUBE 



For a = 1 see | 93t; t he case a = 1 and tp = is the classical Kohn-Morrey 
inequality |100, 101, |126[ (see also [Q). The usual proof of the existence theorem 
is based on a variant of (^ with a = 1 and with different exponential weights (p in 
the different L^q spaces; see for an elegant implementation of this approach. 
Here we will give an argument that has not appeared explicitly in the literature: 
we take ip = and make a good choice of a. 

Suppose that is a pseudoconvex domain: this means that the complex Hessian 
of the defining function p is a non-negative form on the vectors in the complex 
tangent space. Consequently, the boundary integral in (|^) is non-negative. In 
particular, taking a to be identically equal to 1 gives that 

n 

(8) \\du\\' + \\d*ur>Y,'j2\\duj/d-z,r, 

J 3 = 1 

SO the bar derivatives of u are always under control. 

If we replace a by 1 — e**, where b is an arbitrary twice continuously differentiable 
non-positive function, then after applying the Cauchy-Schwarz inequality to the 
term in involving first derivatives of a, we find 

(9) \\V~aduf + \\V~ad*u\\^ > T.' ^T^"^'^"^^ " We'^'d^^- 
Since a + e'' = 1 and a < 1 , it follows that 

(10) \\^uf + Wuf>Y^ / ^''a^"^"^^'^^'^^ 

for every twice continuously differentiable non-positive function h. Notice that this 
inequality becomes a strong one if there happens to exist a bounded plurisubhar- 
monic function h whose complex Hessian has large eigenvalues. (This theme will 
recur later on: see the discussion after Theorem ^ and the discussion of prop- 
erty (P) in section ||.) 

In particular, let p be a point of il, and set h{z) = — 1 + |z — pp/D^, where D is 
the diameter of the bounded domain 17. The preceding inequality then implies the 
fundamental estimate 

(11) \\uf <—{\\duf + \\d*uf). 

q 

Although this estimate was derived under the assumption that u is continuously 
differentiable on the closure il, it holds by density for all square-integrable forms u 
that are in the intersections of the domains of d and d . We also assumed that the 
boundary of f2 is smooth enough to permit integration by parts. Estimate ([ll] ) is 
equivalent to every form in L^q (fl) admitting a representation a.s dv + d w with 
< (Z)^e/g)||u||^. The latter property carries over to arbitrary bounded 
pseudoconvex domains by exhausting a nonsmooth Q by smooth ones, and therefore 
so does inequality ([ll[) . 

Once estimate (^l| is in hand, the proof of Theorem |l| fol lows from standard 
Hilbert space arguments; see, for example, pp. 164-165] or [|l38| , §2]. The latter 
paper also shows the existence of the 9-Neumann operator Nq on (ker9)^. 
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3. Regularity on general pseudoconvex domains 

A basic question is whether one can improve Theorem ^ to get regularity esti- 
mates in Sobolev norms: ||-/Vw||s < C||u||s, \\d Nu\\s < C||u||s, ||9AfM||s < CljuH^. 
If such estimates were to hold for all positive s, then Sobolev's lemma would imply 
that the 9-Neumann operator N (together with d N and dN) is continuous in the 
space C°°{fl) of functions smooth up to the boundary. 

At first sight, it appears that one ought to able to generalize the fundamental 
estimate ( |ll| ) directly to an estimate of the form \\u\\s < C{\\du\\s + \\d u||s), 
simply by replacing u by a derivative of u. This naive expectation is erroneous: 
the difficulty is that not every derivative of a form u in the domain of d is again 
in the domain of d . The usual attempt to overcome this difficulty is to cover the 
boundary of O with special boundary charts p. 33] in each of which one can 
take a frame of tangential vector fields that do preserve the domain of d . Since 
such vector fields have variable coefficients, they do not commute with either d 
or d , and so one needs to handle error terms that arise from the commutators. 

In subsequent sections, we will discuss various hypotheses on the domain Q 
that yield regularity estimates in Sobolev norms. In this section, we discuss firstly 
some completely general results on smoothly bounded pseudoconvex domains and 
secondly some counterexamples. 

It is an observation of J. J. Kohn and his school that the i9-Neumann problem 
is always regular in W'^{fl) for a sufficiently small positive e. 

Proposition 3. Let fl be a bounded pseudoconvex domain in C" with class C°° 
boundary. There exist positive e and C (both depending on fl) such that \\Nu\\i^ < 
C\\u\\f, \\d* Nu\\e < C||u||e, and \\dNu\\^ < C||w||e for every {0,q) form u (where 
l<q<n). 

A proof seems never to have appeared in print, but the idea is very simple. Since 
the commutator of a differential operator of order e with d or d is again an operator 
of order e, but with a coefficient bounded by a constant times e, error terms can be 
absorbed into the main term when e is sufficiently small. 

Theorem 4. Let Q be a bounded pseudoconvex domain in C" with class C°° bound- 
ary. Fix a positive s. There exists a T (depending on s and such that for every t 
larger than T , the weighted d-Neumann problem for the space LJq e~*'^' dV{z)) 

is regular in (SI). In other words, Nt, df.Nt, and dNt are continuous in W^*(S1). 

Moreover, if f is a d-closed {0,q) form with coefficients in C°^{Q), then there 
exists a form u with coefficients in C°°(ri) such that du — f . 

This fundamental result on continuity of the weighted operators is due to Kohn 
|102| . It says that one can always have regularity for the 9-Neumann problem up 
to a certain number of derivatives if one is willing to change the measure with 
respect to which the problem is defined. The idea of the proof is to apply Proposi- 
tion|witha= 1 and Lp{z) ^ t\z\^ to obtain \\e-^\^\^ /"^uf < Ct-W\e-*^'^^ /"^duf + 
||e^*l^l When t is sufficiently large, the factor t^^ makes it possible to 

absorb error terms coming from commutators (see the sketch of the proof of Theo- 
rem ^ below for the ideas of the technique). The resulting a priori estimates are 
valid under the assumption that the left-hand sides of the inequalities are known 
to be finite; Kohn completed the proof by applying the method of elliptic regular- 
ization ]10S{ (see also the remarks after Theorem ^ below) . 
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Via a Mittag-Leffler argument ([103, p. 230], argument attributed to Hormander), 
one can deduce solvability of the equation du ~ f in the space C°°{^1) (but the 
solution will not be the canonical solution orthogonal to the kernel of d). With 
some extra care, the solution operator can be made linear, and also continuous 



from Wf^^,^Jn) n kerd to WI'q JQ) for every positive s and e |139 . It is un- 



known whether or not there exists a linear solution operator for d that breaks even 
at every level in the Sobolev scale. Solvability with Sobolev estimates (with a loss 
of three derivatives) has recently been obtained for domains with only boundary 



by S. L. Yie [ISl] 



Given any solution of the equation du — f, one obtains the canonical solution 
by subtracting from u its projection onto the kernel of d. In view of Kohn's result 
above, it is natural to study the regularity properties of the projection mapping. We 
denote the orthogonal projection from L^q q)i^) ^^^'^ kerc? by P,; when q = 0, this 
operator is the Bergman projection. A direct relation between the Bergman projec- 
tion and the 9-Neumann operator is given by Kohn's formula Pq = Id — 9 Nq+id 
for < q < n. It is evident that if the (9-Neumann operator iV^+i is continuous in 
C°°{fl), then so is Pq. The exact relationship between regularity properties of the 
i9-Neumann operators and the Bergman projections was determined in p5| . 

Theorem 5. Let Q be a bounded pseudoconvex domain in C" with class C°° bound- 
ary. Fix an integer q such that I < q < n. Then the d-Neumann operator Nq is 
continuous on ^^{fl) if and only if the projection operators Pq~i, Pq, and Pq+i 
are continuous on the corresponding C°°{n) spaces. The analogous statement holds 
with the Sobolev space W''^{fl) in place ofC°°(Cl). 

In view of the implications for boundary regularity of biholomorphic and proper 
holomorphic mappings |ll|, 1^, ||, |l|, |l|, ^ |6) , regularity in C°°{Ti) is a key 
issue. 

For some years there was uncertainty over whether the Bergman projection op- 
erator Pq of every bounded domain in C" with C°° smooth boundary might be 
regular in the space C°°(il). Barrett [|| found the first counterexample, motivated 
by the so-called "worm domains" of Diederich and Fornaess . In his example, 
for every p > 2 there is a smooth, compactly supported function whose Bergman 
projection is not in Lp{Q). In Barrett and Fornaess constructed a counterexam- 
ple even more closely related to the worm domains. Although the worm domains 
are smoothly bounded pseudoconvex domains in C^, these counterexamples are 
not pseudoconvex. Subsequently, Kisclman showed that pseudoconvex, but 
nonsmooth, truncated versions of the worm domains have irregular Bergman pro- 
jections. 

Later Barrett (see for a generalization) used a scaling argument together 
with computations on piecewise Levi-flat model domains to show that the Bergman 
projection of a worm domain must fail to preserve the space W^{^1) when s is 
sufhciently large. In view of Theorem ||, the i9-Neumann operator A^i also fails to 
preserve W^^ i)(f^)- This result left open the possibility of regularity in C°^{fl). 
Finally the question was resolved by Christ |Q, as follows. 

Theorem 6. For every worm domain, the Bergman projection operator Pq and the 
d-Neumann operator Ni fail to be continuous on C°°(fl) and Cf^ ^^^{fl). 
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Christ's proof is delicate and indirect. Roughly speaking, he shows that the 
9-Neumann operator does satisfy for most values of s an estimate of the form 
||A^iu||s < C||u||s for all u for which Niu is known a priori to lie in C^^-^{il). If 

Ni were to preserve C^-^^{Q), then density of ^-^{il) in W^^ would imply 

continuity of A^i in W^^ contradicting Barrett's result. 

The obstruction to continuity in ^{il.) for every s on the worm domains is 
a global one: namely, the nonvanishing of a certain class in the first De Rham 
cohomology of the annulus of weakly pseudoconvex boundary points (this class 
measures the twisting of the boundary at the annulus; for details, see Theorem |l5|) . 
For smoothly bounded domains fl, it is known that for each fixed s there is no local 
obstruction in the boundary to continuity in ^{il) 

For all domains fl where continuity in C°°(il) is known, one can actually prove 
continuity in W^ifl) for all positive s. This intriguing phenomenon is not un- 
derstood at present. (The corresponding phenomenon docs not hold for partial 
differential operators in general: see the discussion in section 3 of Christ's article 



56 1 in these proceedings.) 

Although regularity of the 9-Neumann problem in C°°{n) is known in large 
classes of pseudoconvex domains (see sections the example of the worm do- 

mains shows that regularity sometimes fails. At present, necessary and sufficient 
conditions for global regularity of the i9-Neumann operator and of the Bergman 
projection are not known. 

4. Domains of finite type 
Historically, the first major development on the 9-Neumann problem was its 



solution by Kohn |100, 101| for strictly pseudoconvex domains. A strictly pseu- 
doconvex domain can be defined by a strictly plurisubharmonic function, so by 
taking a = 1 and (ys = in (|^) and keeping the boundary term we find that 

\\duf + \\d > C\\ 

^11 L2(bO)' Roughly speaking, this inequality says that we 
have gained half a derivative, since the restriction map W^lbV,) is 

continuous when s > 0. This gain is half of what occurs for an ordinary elliptic 
boundary-value problem, so we have a "subelliptic estimate." 

Theorem 7. Let be a bounded strictly pseudoconvex domain in C" with class 
C°° boundary. If 1 < q < n, then for each non-negative s there is a constant C 
such that the following estimates hold for every (0, q) form u: 

< C(||9u||s -I- u||s) if u ^ domi9 n dom9*, 

(12) < C\\u\\s, 

\\^N,u\l+^^+\\^*N,u\\,^^^<C\\u\l. 

The standard reference for the proof of this result is |Q (where the theory is 
developed for almost complex manifolds); see also [ |113[ . The estimates can be 
localized, as in Theorem ^ below. 

A key technical point in the proof of Theorem |^ is that after establishing the 
estimates under the assumption that the left-hand side is a priori finite, one then 
has to convert the a priori estimates into genuine estimates, in the sense that the 
left-hand side is finite when the right-hand side is finite. Kohn's original approach 



was considerably simplified in [108| in a very general framework, via the elegant 
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device of "elliptic regularization." The idea of the method is to add to □ an elliptic 
operator times e (thereby obtaining a standard elliptic problem) , to prove estimates 
independent of e, and to let e go to zero. (The analysis Christ used Q to prove 
Theorem |^ shows that indeed a priori estimates cannot always be converted into 
genuine estimates. For this phenomenon in the context of the Bergman projection, 
see (ISTII.) Another interesting approach to the proof of Theorem]^ was indicated by 



Morrey |127 



A number of authors (see g9[ and their references) have refined the results for 
strictly pseudoconvex domains in various ways, such as estimates in other function 
spaces and anisotropic estimates. In particular, N gains two derivatives in complex 
tangential directions; this gain results from the bar derivatives always being under 
control (sec (||)). Integral kernel methods have a.lso b een developed succ essfully on 
strictly pseudoconvex domains; see |9|, 115, 116, 133, 135, 136] and their 

references. 

The gain of one derivative for the 9-Neumann operator Ni in Theorem Q is 
sharp, and the domain is necessarily strictly pseudoconvex if t his e stimate holds. 
For discussion of this point, see ||, §|, §111.2], |9|, §3.2], and j|ll], §4]. 

More generally, one can ask when the 9- Neumann operator gains some fractional 
derivative. One says that a subcUiptic estimate of order e holds for the 9-Neumann 
problem on (0, q) forms in a neighborhood [/ of a boundary point zq of a pseudo- 
convex domain in C" if there is a constant C such that 



(13) 



\u\\l<C{\\du\\l + \\du\\l) 



for every smooth (0, q) form u that is supported in J7n 51 and that is in the domain 
of d . The systematic study of subelliptic estimates in 1 10^ ] provides the following 
"pseudolocal estimates." 

Theorem 8. Let Q be a bounded pseudoconvex domain in C" with class C°° bound- 
ary. Suppose that a subelliptic estimate (|l^) holds in a neighborhood U of a bound- 
ary point Zq. Let xi o.nd X2 be smooth cutoff Junctions supported in U with X2 iden- 
tically equal to 1 in a neighborhood of the support ofxi- For every non-negative s, 
there is a constant C such that the d-Neumann operator Nq and the Bergman pro- 
jection Pq satisfy the estimates 

\\XlNqU\\s+2t < C{\\x2u\\s + \\u\\o), I < q < n, 

(14) \\xid*Nqu\\s+, + \\xidNqu\\s+e<C{\\x2u\\s + \\u\\o), I < q < n, 

WxiPquWs < C{\\x2u\\s + llujlo), 0<q<n. 

Consequently, if a subelliptic estimate ( p^ holds in a neighborhood of every 
boundary point of a smooth bounded pseudoconvex domain D, in C", then the 
Bergman projection is continuous from W^^ {Q) to itself, and the i9-Neumann 

operator is continuous from W^^Jfl) to Wf^-^{Q). 

In a sequence of papers ]Q, |8[ |6^, Kohn's students David Catlin 

and John D'Angelo resolved the question of when subelliptic estimates hold in 
a neighborhood of a boundary point of a smooth bounded pseudoconvex domain 
in C". The necessary and sufficient condition is that the point have "finite type" 
in an appropriate sense. We briefly sketch this work; for details, consult the above 
papers as well as ||, |6[ |7|, |lO|, |lO|, ^ |lO|] and the survey by D'Angelo 
and Kohn ]p^ in these proceedings. 
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The simplest obstruction to a subelliptic estimate is the presence of a germ of 
an analytic variety in the boundary of a domain. Indeed, examples show that local 
regularity of the 9- Neumann problem fails when there are complex varieties in the 
boundary; see p3| , iTOf . If the boundary is real-analytic near a point, then the ab- 
sence of germs of (/-dimensional complex-analytic varieties in the boundary near the 
point is necessary and sufficient for the existence of a subelliptic estimate on (0, q)- 
forms |l05[ . This was first proved by combining a sufficient condition from Kohn's 
theory of ideals of subelliptic multipliers [105| with a theorem of Diederich and 
Fornaess on analytic varieties. Moreover, Diederich and Fornaess showed that a 
compact real-analytic manifold contains no germs of complex-analytic varieties of 
positive dimension, so subelliptic estimates hold for every bounded pseudoconvex 
domain in C" with real-analytic boundary. 

The first positive results in the C°° category were established in dimension two. 
A boundary point of a domain in is of finite type if the boundary has finite 
order of contact with complex manifolds through the point; equivalently, if some 
finite-order commutator of complex tangential vector fields has a component that 
is transverse to the complex tangent space to the boundary. If m is an upper 
bound for the order of contact of complex manifolds with the boundary, then a 
subelliptic estimate jl^ ) holds with e = 1/m. For these results, see 105 1; for 
the equivalence of the two notions of finite type, see |p9| . For pseudoconvex domains 
of finite type in dimension two, sharp estimates for the 9-Neumann problem are 
now known in many function spaces (see |5^ and their references). 

In higher dimensions, it is no longer the case that all reasonable notions of finite 
type agree; for relations among them, see D'Angelo's notion of finite type 

has turned out to be the right one for characterizing subelliptic estimates for the 
9-Neumann problem. His idea to measure the order of contact of varieties with a 
real hypersurface M in C" at a point zq is to fix a defining function p for Af and 
to consider the order of vanishing at the origin of p o f , where / is a nonconstant 
holomorphic mapping from a neighborhood of the origin in C to C" with /(O) — zo- 
Since the variety that is the image of / may be singular, it is necessary to normalize 
by dividing by the order of vanishing at the origin of / ( • ) — zq ■ The supremum 
over all / of this normalized order of contact of germs of varieties with M is the 
D'Angelo 1-type of zq. 

Theorem 9. The set of points of finite 1-type of a smooth real hypersurface M 
in C" is an open subset of M , and the 1-type is a locally bounded function on M. 

This fundamental result of D'Angelo |62j is remarkable, because the 1-type may 
fail to be an upper semi-continuous function (see j6^, p. 136] for a simple example). 
The theorem implies that if every point of a bounded domain in C" is of finite 
1-type, then there is a global upper bound on the 1-type. 

For higher-dimensional varieties, there is no canonical way that serves all pur- 
poses to define the order of contact with a hypersurface. Catlin defined a 
quantity Dq{zQ) that measures the order of contact of g-dimensional varieties in 
"generic" directions (and Di agrees with D'Angelo's 1-type). Catlin's fundamental 
result is the following. 

Theorem 10. Let Q be a bounded pseudoconvex domain in C" with class C°° 
boundary. A subelliptic estimate for the d-Neumann problem on (0, q) forms holds 
in a neighborhood of a boundary point zq if and only if Dq{zo) is finite. The e in 
the subelliptic estimate ( |l3| ) satisfies e < l/Dq{zo). 
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Catlin proved the necessity of finite order of contact, together with the upper 
bound on e, in Q| (see also and the sufficiency in |Q. Cathn's proof of 

sufficiency has two parts. His theory of multitypes [4^ implies the existence of a 
stratification of the set of weakly pseudoconvex boundary points. The stratification 
is used to construct families of bounded plurisubharmonic functions whose complex 
Hessians in neighborhoods of the boundary have eigenvalues that blow up like 
inverse powers of the thickness of the neighborhoods. Such powers heuristically act 
like derivatives, and so it should be plausible that the basic inequality ( p^ ) leads 
to a subelliptic estimate (p^. 

It is unknown in general how to determine the optimal value of e in a subelliptic 
estimate in terms of boundary data. For convex domains of finite type in C", the 
optimal e i n a s ubelliptic estimate for (0, 1) forms is the reciprocal of the D'Angelo 
1-type |85 , 124 ; this is shown by a direct construction of bounded pluri subh armonic 



functions with suitable Hessians near the boundary. McNeal proved 124 that for 
convex domains, the D'Angelo 1-type can be computed simply as the maximal order 
of contact of the boundary with complex lines. (There is an elementary geometric 
proof of McNeal's result in and an analogue for Reinhardt domains in It 
is clear that in general, the best e cannot equal the reciprocal of the type, simply 
because the type is not necessarily upper semi-continuous. For more about this 
subtle issue, see [|5[ |6[ ^ \n\ . 

5. Compactness 

A subelliptic estimate (|l^ ) implies, in particular, that the 9-Neumann operator 
is compact as an operator from L^g q){^) to itself. This follows because the embed- 
ding from {fl) into L^p (fl) is compact when is bounded with reasonable 



boundary, by the Rellich-Kondrashov theorem (see, for example, |149, Prop. 25.5]). 
One might think of compactness in the (9-Neumann problem as a limiting case of 
subellipticity as e ^ 0. 

The following lemma reformulates the compactness condition. 

Lemma 11. Let Q be a bounded pseudoconvex domain in C", and suppose that 
1 < q < n. The following statements are equivalent. 

1. The d-Neumann operator Nq is compact from q)i^) itself. 

2. The embedding of the space domc?n dom9 , provided with the graph norm 
u I— > ||9u||o -f \\d w||o; into ,j)(^) '■^ compact. 

3. For every positive e there exists a constant Ce such that 

(15) \\u\\l<e{\\du\\l + \\d*n\\l) + C4\u\\\ 

when u G dom d n dom d . 

Statement ^ is called a compactness estimate for the (9-Neumann problem. Its 
equivalence with statement ||is in [108, Lemma 1.1]. The equivalence of statement 1 



with statements ^ and ^ follows easily from the theory discussed in section 2 
and the compactness of the embedding Lj(, q){^) ^ 

In view of Theorem ||, it is a reasonable guess that compactness in the d- 
Neumann problem implies global regularity of the 9-Neumann operator in the sense 



that Nq maps q){^) into itself. Work of Kohn and Nirenberg |108] shows that 
this conjecture is correct. 
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Theorem 12. Let Q be a bounded pseudoconvex domain in C" with class C°° 
boundary, and suppose 1 < q < n. If a compactness estimate (|l5|) holds for 
the d-Neumann problem on (0, q) forms, then the d-Neumann operator Nq is a 
compact (in particular, continuous) operator from W^(og)(^) ^'^^o itself for every 
non-negative s. 

It suffices to prove the result for integral s, as the intermediate cases then follow 
from standard interpolation theorems |132| . We sketch the argument for s = 1 , 
which illustrates the method. To prove the compactness of the 9-Neumann operator 
in '^^1^ establish the (a priori) estimate ||-/Vgu||f < e||u||f + Ce||M|jg for 

arbitrary positive e under the assumption that u and NqU are both in C°°(r2). 

First we show that the compactness estimate ( p^ ) lifts to 1-norms: namely, 
ll^llf < e(||9u||i + u\\\) + Cf_\\u\\'^_i for smooth forms u in dome? (with a new 
constant Cc). In a neighborhood of a boundary point, we complete dp to an or- 
thogonal basis of (0, 1) forms and choose dual vector fields. To estimate tangential 
derivatives of u, we apply (^5|) to these derivatives (valid since they preserve the 
domain of 9 ). We then commute the derivatives with d and d , which gives an 
error term that is of the same order as the quantity on the left-hand side that 
we are trying to estimate, but multiplied by a factor of e. We also need to es- 
timate the normal derivative of u, but since the boundary is noncharacteristic 
for the elliptic complex d (B d , the normal derivative of u can be expressed in 
terms of du, d u, and tangential derivatives of u. Summing over a collection of 
special boundary charts that cover the boundary, and using interior elliptic reg- 
ularity to estimate the norm on a compact set, we obtain an inequality of the 
form \\u\\l < Ae{\\du\\i + \\d*u\\l + \\u\\f) + Bi\\du\\l + \\d*u\\l + where the 

constants A and B are independent of e. We can use the standard interpolation 
inequality ||/||s < e||/||s+i + C£||/||s-i to absorb terms into the left-hand side when 
e is sufficiently small. 

The lifted compactness estimate together with the boundedness of the d- 
Neumann operator implies 

(16) \\Nqu\\j < e{\\dNqu\\l + \\d*Nqu\\j) + Ce\\u\\l 

Working as before in special boundary charts, we commute derivatives and integrate 
by parts on the right-hand side to make dd NqU -f d dNqU = u appear (see [ 107 , 



p. 140], 1 35, p. 31]). Keeping track of commutator error terms and applying the 
Cauchy-Schwarz inequality, we find 

(17) 

\\dNqU\\\ + \\d*NqU\\l < A{\\NqU\\l\\u\\l + \\u\\l + {\\dNqU\\l + \\d* NqU\\l)\\NqU\\l) 

for some constant A. Consequently \\dNqu\\l + \\d* Nqu\\l < B{\\Nqu\\l + \\u\\l) for 
some constant B. Combining this with ( p^ gives the required a priori estimate 

(18) \\Nqu\\l<e\\u\\l + CM\l 



Kohn and Nirenberg 1 108 ] developed the method of elliptic regularization (described 
above after Theorem^ to convert these a priori estimates into genuine ones. 

There is a large class of domains for which the 9-Neumann operator is compact 
|46| 14C|. In |4^, Catlin introduced "property (P)" and showed that it implies a 



compactness estimate ([l5| ) for the 9-Neumann problem. A domain f2 has prop- 
erty (P) if for every positive number M there exists a plurisubharmonic function A 
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in C°°(ri), bounded between and 1, whose complex Hessian has all its eigenvalues 
bounded below by M on 



That property (P) implies a compactness estimate ( p^ ) follows directly from (10) 
and interior elliptic regularity. 

It is easy to see that the existence of a strictly plurisubharmonic defining func- 
tion implies property (P), so strictly pseudoconvex domains satisfy property (P). 
So do pseudoconvex domains of finite type: this was proved by Catlin ||4q| as a 
consequence of his and D'Angelo's analysis of finite type boundaries |62| . 

Property (P) is, however, much more general than the condition of finite type. 
For instance, it is easy to see that a domain that is strictly pseudoconvex except for 
one infinitely fiat boundary point must have property (P). More generally, prop- 
erty (P) holds if the set of weakly pseudoconvex boundary points has Hausdorff 
two-dimensional measure equal to zero 14C]. Sibony |14C] made a systematic 



study of the property (under the name of "B-regularity" ) . In particular, he found 
examples of B-regular domains whose boundary points of infinite type form a set 
of positive measure. 

It is folklore that an analytic disc in the boundary of a pseudoconvex domain 
in obstructs compactness of the 9-Neumann problem: this can be proved by 
an adaptation of the argument used in p3| , [79[ to show (in any dimension) that 
analytic discs in the boundary preclude hypoellipticity of d. In higher dimensions, 
tamely embedded analytic discs in the boundary obstruct compactness, but the 



general situation seems not to be understood; see [112, 119| for a discussion of 
some interesting examples. Salinas found an obstruction to compactness phrased in 
terms of the C*-algebra generated by the operators of multiplication by coordinate 



functions (see the survey |137| and its references). 

In view of the maximum principle, property (P) excludes analytic structure from 
the boundary: in particular, the boundary cannot contain analytic discs. However, 
the absence of analytic discs in the boundary does not guarantee property (P) 
|140| p. 310], although it does in the special cases of convex domains and complete 
Rcinhardt domains |140| , Prop. 2.4]. 

It is not yet understood how much room there is between property (P) and 
compactness. Having necessary and sufficient conditions on the boundary of a 
domain for compactness of the 9-Neumann problem would shed considerable light 
on the interactions among complex geometry, pluripotcntial theory, and partial 
differential equations. 

6. The vector field method 

In the preceding section, we saw that the 9-Neumann problem is globally regular 
in domains that support bounded plurisubharmonic functions with arbitrarily large 
complex Hessian at the boundary. Now we will discuss a method that applies, for 
example, to domains admitting defining functions that are plurisubharmonic on the 
boundary. The method is based on the construction of certain vector fields that 
almost commute with d. 

We begin with some general remarks about proving a priori estimates of the 
form IjA'qulls < CIImUs and < CHuHs in Sobolev spaces for the 9-Neumann 
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operator and the Bergman projection. Firstly, all the action is near the boundary. 
This is clear for the Bergman projection on functions, because the mean-value 
property shows that every Sobolev norm of a holomorphic function on a compact 
subset of a domain is dominated by a weak norm on the whole domain (for instance, 
the norm) . The corresponding property holds for the c)-Neumann operator due 
to interior elliptic regularity. 

Secondly, the conjugate holomorphic derivatives d/dzj are always under con- 
trol. This is obvious for the case of the Bergman projection Pq on functions (since 
holomorphic functions arc annihilated by anti- holomorphic derivatives), and the in- 
equality (^ shows that anti-holomorphic derivatives are tame for the i9-Neumann 
problem. 

Thirdly, differentiation by vector fields whose restrictions to the boundary lie in 
the complex tangent space is also innocuous. Indeed, integrating by parts turns 
tangential vector fields of type (1, 0) into vector fields of type (0, 1), which are tame, 
plus lower-order divergence terms |3^, formula (3)]. 

Thus, we only need to estimate derivatives in the complex normal direction near 
the boundary. Moreover, since the bar derivatives are free, it will do to estimate 
either the real part or the imaginary part of the complex normal derivative. That 
is, we can get by with estimating either the real normal derivative, or a tangential 
derivative that is transverse to the complex tangent space. 

A simple application of these ideas shows, for example, that the Bergman projec- 
tion Pq on functions for every bounded Reinhardt domain Vt in C" with class C°° 



boundary is continuous from M^''(il) to itself for every positive integer s 143 1. 
Indeed, the domain is invariant under rotations in each variable, so the Bergman 
projection commutes with each angular derivative d/dOj. At every boundary point, 
at least one of these derivatives is transverse to the complex tangent space, so 
llPo^^lli < CE;UI1(W)^'ou|1o = CY.%^\\Po{du/dB,)\\^ < C'Wuh. Higher 
derivatives are handled analogously. A similar technique proves global regularity 
of the 9-Neumann operator on bounded pseudoconvex Reinhardt domains ^ . 

Thus, the nicest situation for proving estimates in Sobolev norms for the d- 
Neumann operator is to have a tangential vector field, transverse to the complex 
tangent space, that commutes with the (9-Neumann operator, or what is nearly the 



same thing, that commutes with d and d . (This method is classical |6^, |70|, 109].) 
Actually, it would be enough for the commutator with each anti-holomorphic de- 
rivative d/dzj to have vanishing (1, 0) component in the complex normal direction. 
However, work of Derridj Theoreme 2.6 and the remark following it] shows that 
no such field can exist in general. 

If we have a real tangential vector field T, transverse to the complex tangent 
space, whose commutator with each d/dzj has (1,0) component in the complex 
normal direction of modulus less than e, then we get an estimate of the form 
WT^NquWo < As{\\u\\s + e\\Nqu\\s) + Cs,t\\u\\q. If the field T is normalized so that 
its coefficients and its angle with the complex tangent space are bounded away 
from zero, then HT^A^^uHo controls ||A'^gw||s (independently of e), so we get global 
regularity of Nq up to a certain level in the Sobolev scale. (By making estimates 
uniformly on a sequence of interior approximating strongly pseudoconvex domains, 
we can convert the a priori estimates to genuine ones.) Moreover, it suffices if T is 
approximately tangential in the sense that its normal component is of order e. (This 
idea comes from work of Barrett [||; see the proof of Theorem ^) If we can find 
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a sequence of such normahzed vector fields corresponding to progressively smaller 
values of e, then the 9-Neumann problem is globally regular at every level in the 
Sobolev scale. Because of the local regularity at points of finite type, the vector 
fields need exist only in (progressively smaller) neighborhoods of the boundary 
points of infinite type. In other words, we have the following result (where the 
imaginary parts of the correspond to the vector fields described above) ^ . 

Theorem 13. Let Q be a bounded pseudoconvex domain in C" with class C°° 
boundary and defining function p. Suppose there is a positive constant C such that 
for every positive e there exists a vector field Xf of type (1,0) whose coefficients are 
smooth in a neighborhood Uc in C" of the set of boundary points of Q of infinite 
type and such that 

1. I argXepI < e on U^, and moreover C^^ < \Xi,p\ < C on U^, and 

2. when I < j < n, the form dp applied to the commutator [X^, d/dzj] has 
modulus less than e onU^. 

Then the d-Neumann operators Nq (for I < q < n) and the Bergman projections Pq 
(for < q < n) are continuous on the Sobolev space W^^ ^^^^^ s > 0. 

For a simple example in which the hypothesis of this theorem can be verified, 
consider a ball with a cap sliced off by a real hyperplane, and the edges rounded. 
The normal direction to the hyperplane will serve as X^ (the C/g being shrinking 
neighborhoods of the flat part of the boundary), so the i9-Neumann operator for 
this domain is continuous at every level in the Sobolev scale. 

Indeed, the hypothesis of Theorem |l^ can be verified for all convex domains. 
(The regularity of the 9-Neumann problem for convex domains in dimension two 
was obtained independently by Chen using related ideas.) More generally, the 
theorem applies to domains admitting a defining function that is plurisubharmonic 
on the boundary We state this as a separate result and sketch the proof. 

(Continuity in W^/'^{n) in the presence of a plurisubharmonic defining function 
was obtained earlier by Bonami and Charpentier 0-) 

Theorem 14. Let Q be a bounded pseudoconvex domain in C" with class C'°° 
boundary. Suppose that fl has a C°° defining function p that is plurisubharmonic 
on the boundary: k=ii^'^ P/^^J^^k)wjWk > for all z & bfl and all w € C". 
Then for every positive s there exists a constant C such that for all u G W^q (fi) 

we have 

\\Nqu\l<C\\u\\s, l<g<n, 
\\Pqu\\,<C\\u\\s, Q<q<n. 

Pseudoconvexity says that on the boundary, ^.^i{d^p/dzjdzk)wj'Wk > for 
vectors w in the complex tangent space: those vectors for which jyj=i i9p/dzj)wj — 
0. The hypothesis of the theorem is that on the boundary, the complex Hessian 
of p is non-negative on all vectors, not just complex tangent vectors. (There are 
examples of pseudoconvex domains, even with real-analytic boundary, that do not 
admit such a defining function even locally B P, ||.) We now sketch how this 
extra information can be used to construct the special vector fields needed to invoke 
Theorem H 

The key observation is that for each j, derivatives of dp/dzj of type (0, 1) in direc- 
tions that lie in the null space of the Levi form must vanish. Indeed, if d/dzi (say) 
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is in the null space of the Levi form at a boundary point p, then d^p/dzidzi{p) — 0, 
but since the matrix d^p/dzjdzk{p) is positive semidefinite, its whole first column 
must vanish. (It was earlier observed by Noell [129| that the unit normal to the 
boundary of a convex domain is constant along Levi- null curves.) 

To construct the required global vector field, it will suffice to construct a vector 
field whose commutator with each complex tangential field of type (1,0) has van- 
ishing component in the complex normal direction at a specified boundary point p. 
Indeed, these components will be bounded by e in a neighborhood of p by conti- 
nuity, and we can use a partition of unity to patch local fields into a global field. 
(Terms in the commutator coming from derivatives of the partition of unity cause 
no difficulty because they are complex tangential.) It is easy to extend the field 
from the boundary to the inside of the domain to prescribe the proper commutator 
with the complex normal direction. 

Suppose that dp/dzn{p) ^ 0. We want to correct the field {dp/ dzn)~^ {d / dzn) by 
subtracting a linear combination of complex tangential vector fields so as to adjust 
the commutators. Since the Levi form may have some zero eigenvalues at p, we 
need a compatibility condition to solve the resulting linear system. The observation 
above that type (0, 1) derivatives in Levi-nuU directions annihilate dp/dzn at p is 
precisely the condition needed for solvability. For details of the proof, see |Q . 

Kohn has found a new proof and generalization of T heorem According 
to a theorem of Diederich and Forna^ss (see also |134|), a smooth bounded 
pseudoconvex domain admits a defining function such that some (small) positive 
power of its absolute value is plurisuper harmonic inside f2; let 5 denote the supre- 
mum of such exponents. Kohn showed that there is a constant A such that the 
9-Neumann problem is regular in W'^{Vl) when (1 — 5)sA'^ < 1. (This result also 
contains Proposition ||.) 

Theorem |l^ applies to other situations besides the one described in Theorem ^ 
For instance, it is possible to construct the vector fields on pseudoconvex domains 
that are regular in the sense of Diederich and Fornaess |^ and Catlin Q| . (This 
gives no new theorem, however, since the c)-Neumann problem is known to be 
compact on such domains ; nor does it give a simplified proof of global regularity 
in the finite type case, since the construction of the vector fields still requires Catlin's 
stratification of the set of weakly pseudoconvex points ]4^.) 

As mentioned in section |^, global regularity for the 9-Neumann problem breaks 
down on the Diederich- Fornaess worm domains. On those domains, the set of weakly 
pseudoconvex boundary points is precisely an annulus, and it is possible to compute 
directly that the vector fields specified in Theorem |l^ cannot exist on this annulus. 

For domains of this kind, where the boundary points of infinite type form a 
nice submanifold of the boundary, there is a natural condition that guarantees the 
existence of the vector fields needed to apply Theorem |l^. Following the notation 
of we let 77 denote a purely imaginary, non- vanishing one- form on the 

boundary bQ that annihilates the complex tangent space and its conjugate. Let 
T denote the purely imaginary tangential vector field on bfl orthogonal to the 
complex tangent space and its conjugate and such that 'ri{T) = 1. Up to sign, the 
Levi form of two complex tangential vector fields X and Y is ri{[X, Y]). The (real) 
one-form a is defined to be minus the Lie derivative of 77 in the direction of T: 

(20) a = -CtV- 
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One can show §2] that if M is a submanifold of the boundary whose real tangent 
space is contained in the nuU space of the Levi form, then the restriction of the 
form a to M is closed, and hence represents a cohomology class in the first De Rham 
cohomology H^{M). (In the special case when M is a complex submanifold, this 
closedness corresponds to the pluriharmonicity of certain argument functions, as 
in |], [0 Prop. 3.1], Lemma 1], and Q p. 290].) This class is independent 
of the choice oi rj. If this cohomology class vanishes on such a submanifold M, 
and if M contains the points of infinite type, then the vector fields described in 
Theorem ^ do exist. Thus, we have the following result p9[ . 

Theorem 15. Let Q be a bounded pseudoconvex domain in C" with class C°° 
boundary. Suppose there is a smooth real submanifold M (with or without boundary) 
of foil that contains all the points of infinite type of bQ and whose real tangent space 
at each point is contained in the null space of the Levi form at that point ( under the 
usual identification o/R^" with C"). If the H^{M) cohomology class [a|A/] is zero, 
then the d-Neumann operators Nq (for 1 < q < n) and the Bergman projections Pq 
(for < q < n) are continuous on the Sobolev space II^^q when s > 0. 

On the worm domains, one can compute directly that the class [q;|m] is not 
zero. The appearance of this cohomology class explains, in particular, why an 
analytic annulus in the boundary of the worm domains is bad for Sobolev estimates, 
while an annulus in the boundary of other domains may be innocuous [^7| , and an 
analytic disc is always benign In the special case that n — 2 and M is a 

bordered Riemann surface, Barrett has shown that there is a pluripolar subset 
of H^{M) such that estimates in W'^{Vl) fail for sufficiently large k if [a|Af] lies 
outside this subset When M is a complex submanifold of the boundary, [a|j\/] 
has a geometric interpretation as a measure of the winding of the boundary of 
around M (equivalently, the winding of the vector normal to the boundary). For 
details, see |12[. (In the context of Hartogs domains in C^, see also [M.) 



The constructions of the vector fields (needed to apply Theorem 13) in the 
proofs of Theorems |l^ and |l^ are more closely related than appears at first glance. 
The vector fields can be written locally in the form e^L^ + ^j^j' where 

ii, . . . , L„_i form a local basis for the tangential vector fields of type (1, 0), L„ is the 
normal field of type (1, 0), and h and the Oj are smooth functions. The commutator 
conditions in Theorem |l^ in directions not in the null space of the Levi form can 
always be satisfied by using the Oj to correct the commutators. Computing the 
commutators in the remaining directions leads to the equation = q;|aA(p) at 

points p of infinite type (where M{p) is the null space of the Levi form at p) . The 
above proof of Theorem ^ amounts to showing that (y\j^{p) — when there is a 
defining function that is plurisubharmonic on the boundary, whence /i = gives a 
solution. In Theorem [l^, the hypothesis of the vanishing of the cohomology class 
of a on M allows us to solve for h (on M). 

In general, the points of infinite type need not lie in a "nice" submanifold of the 
boundary. It is not known what should play the role of the cohomology class [a|j\/] 
in the general situation. (Note that the analogue of the property that a\M is closed 
holds in general: da\j^(p) = 0; see §2].) Furthermore, it is not understood 
how to combine the ideas of this section with the pluripotential theoretic methods 
discussed in section 5 (_B-regularity/property (P)). 



GLOBAL REGULARITY OF THE O-NEUMANN PROBLEM 



17 



7. The Bergman projection on general domains 

In pseudoconvex domains, global regularity of the 9-Neumann problem is essen- 
tially equivalent to global regularity of the Bergman projection | |35[ |. In nonpseu- 
doconvex domains, the 9-Neumann operator may not exist, yet the Bergman pro- 
jection is still well defined. Since global regularity of the Bergman projection on 
functions is intimately connected to the boundary regularity of biholomorphic and 
proper holomorphic mappings |2j, |l6[ |l^, ^ |72, 86|, it is interesting to ana- 



lyze the Bergman projection directly, without recourse to the 9-Neumann problem. 
Even very weak regularity properties of the Bergman projection can be exploited 
in the study of biholomorphic mappings ||^, 114\. 



In this section, we survey the theory of global regularity of the Bergman projec- 
tion on general (that is, not necessarily pseudoconvex) domains. 

The first regularity results for the Bergman projection that were obtained with- 
out the help of the 9-Neumann theory are in ||2^, where it is shown that the 
Bergman projection P on functions maps the space C°°{Q) of functions smooth up 
to the boundary continuously into itself when il is a bounded complete Reinhardt 
domain with C°° smooth boundary. 

This result was generalized in ^ to domains with "transverse symmetries." A 
domain is said to have transverse symmetries if it admits a Lie group G of 
holomorphic automorphisms acting transversely in the sense that the map G x 
^ taking {g, z) to g{z) extends to a smooth map G x ^ il, and for each 
point zq S hVL the map g i~> g{zo) of G to bU, induces a map on tangent spaces 
TidG Tf-^{b^) whose image is not contained in the complex tangent space to hVl 
at zq- In other words, there exists for each boundary point zq a one-parameter 
family of automorphisms of whose infinitesimal generator is transverse to the 
tangent space at zq. This class of domains includes many Cartan domains as 
well as all smooth bounded Reinhardt domains; in both cases, suitable Lie groups 
of rotations provide the transverse symmetries H. For domains with transverse 



symmetries, it was observed in [143 that the Bergman projection not only maps 
the space C°°{Q) into itself, but actually preserves the Sobolev spaces. 

More generally, one can obtain regularity results in the presence of a transverse 
vector field of type (1,0) with holomorphic coefficients, even if it does not come 
from a family of automorphisms. David Barrett obtained the following result j^. 

Theorem 16. Let Q be a bounded domain in C" with class G°° boundary and 
defining function p. Suppose there is a vector field X of type (1, 0) with holomorphic 
coefficients in C°°{Q) that is nowhere tangent to the boundary of and such that 
\avgXp\ < 7r/4fc for some positive integer k. Then the Bergman projection on 
functions maps the Sobolev space W''{n) continuously into itself. 

In particular. Theorem implies that there are no local obstructions to reg- 
ularity of the Bergman projection. In other words, any sufficiently small piece of 
G°° boundary can be a piece of the boundary of a domain G whose Bergman pro- 
jection is continuous in W'^{G): indeed, G can be taken to be a small perturbation 
of a ball, and then the radial field satisfies the hypothesis of the theorem. 

Theorem |l^ also applies when k = 1/2 and the boundary is only Lipschitz 
smooth. For example, the hypothesis holds for k = 1/2 when the domain is strictly 



star-shaped. Lempert [114| has exploited this weak regularity property to prove 
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a Holder regularity theorem for biholomorphic mappings between star-shaped do- 
mains with real-analytic boundaries. 

The first step in the proof of Theorem ^ is one we have seen before in section ^ 
namely, it suffices to estimate derivatives of holomorphic functions in a direction 
transverse to the boundary. Thus, to bound ||P/||fc it suffices to bound ||X'^P/||o. 
However, the inner product {X^Pf^X^Pf) is bounded above by a constant times 
\{lP^ X^ P f , X^ P f)\ when ^Lp^ is bounded away from zero. By the hypothesis of 
the theorem, we can take (/s to be a smooth function that equals Xp/Xp near 
the boundary. Wc then replace (p^X^ on the left-hand side of the inner product by 
[ifX — X)^, making a lower-order error (since X annihilates holomorphic functions). 
The point is that {^X — X) is tangential at the boundary, so we can integrate 
by parts without boundary terms, obtaining \ {P f ^ X"^^ P f) \ plus lower-order terms. 
Since X is a holomorphic field, we can remove the Bergman projection operator from 
the left-hand side of the inner product, integrate by parts, and apply the Cauchy- 
Schwarz inequality to get an upper bound of the form C||/||fc||P/||fc. This gives an 
a priori estimate ||-P/||fe < C||/||fe. The estimate can be converted into a genuine 
estimate via an argument involving the resolvent of the semigroup generated by the 
real part of X. For details of the proof, see 1^. 

It is possible to combine such methods with techniques based on pseudocon- 
vexity. Estimates for the Bergman projection and the 9-Neumann operator on 
pseudoconvex domains that have transverse symmetries on the complement of a 
compact subset of the boundary consisting of points of finite type were obtained in 



50| and [g. 

A domain in is called a Hartogs domain if, with each of its points {z,w), it 
contains the circle {(z. Aw) : |A| = 1}; it is complete if it also contains the disc 
{{z^Xw) : |A| < 1}. The (pseudoconvex) worm domains [7^ and the (nonpseudo- 
convex) counterexample domains in ||^, ^ with irregular Bergman projections are 
incomplete Hartogs domains in C'^. It is easy to see that when a Hartogs domain 
in is complete, the obstruction to regularity identified in section |^ cannot occur 
(see §1]). Actually, completeness guarantees that the Bergman projection is 
regular whether or not the domain is pseudoconvex ||3^ . (See for a systematic 
study of the Bergman projection on Hartogs domains in C^.) 

Theorem 17. Let fl be a bounded complete Hartogs domain in with class C°° 
boundary. The Bergman projection maps the Sobolev space W''{^) continuously 
into itself when s > 0. 

The proof again uses different arguments on different parts of the boundary. An 
interesting new twist occurs in that the 9-Neumann operator of the envelope of 
holomorphy of the domain (which is still a complete Hartogs domain) is exploited. 

The Bergman projection is known to preserve the Sobolev spaces ^{^1) in 
all cases in which it is known to preserve the space C°°{il) of functions smooth 
up to the boundary (as is the case for the 9-Neumann operator on pseudoconvex 
domains). It is an intriguing question whether or not this is a general phenomenon. 

We now turn to the connection between the regularity theory of the Bergman 
projection and the duality theory of holomorphic function spaces, which originates 
with Bell . When k is an integer, let A'' (fi) denote the subspace of the Sobolev 
space ly'^(ri) consisting of holomorphic functions, and let A°°{U) denote the sub- 
space of C°°(ri) consisting of holomorphic functions. We may view the Frechet 
space A°°(n) as the projective fimit of the Hilbert spaces A''{Q), and we introduce 
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the notation for the space IJfe^i ^ provided with the inductive hmit 

topology. 

For discussion of some of the technical properties of these spaces of holomorphic 



functions, see |2^, 142 1. In particular, A~°°{Q) is a Montel space, and subsets 
of A^°°{^1) are bounded if and only if they are contained and bounded in some 
A~''{n). The inductive limit structure on A^°°{il) turns out to be "nice" because 
the embeddings A^'^(ri) A~''~^{il.) are compact (as a consequence of Rellich's 
lemma). Functions in A^°^'{fl) can be characterized in two equivalent ways: they 
have growth near the boundary of that is at most polynomial in the reciprocal of 
the distance to the boundary, and their traces on interior approximating surfaces 



bfle converge in the sense of distributions on bfl. See [142| for an elementary 
discussion of these facts. 

The inner product extends to a more general pairing. Harmonic functions 
are a natural setting for this extension. We use the notations h°°{^) and h~°°{il) 
for the spaces of harmonic functions analogous to A°°{^) and 

Proposition 18. Let ^ be a bounded domain in C" with class C°° boundary. For 
each positive integer k there is a constant Ck such that for every square-integrable 
harmonic function f , and every g € C°°(ri), we have the inequality 



(21) 



fg 



n 



<^^fc||/ll-fc||3lk- 



The proof of Proposition follows from the observation that for every g G 
C°°(fi), there is a function gi vanishing to high order at the boundary of ft such 
that the difference g — gi is orthogonal to the harmonic functions. See |2q], [Bll 



Appendix B], and |120| for details; the root idea originates with Bell |l9t] in the 



context of holomorphic functions. Alternatively, Proposition [18^ can be derived from 



elementary facts about the Dirichlet problem for the Laplace operator [142|. 

Because the square-integrable harmonic functions are dense in h~°°{n), it follows 
fro m^pl] ) that the pairing extends by continuity to a pairing (/, g) on h^°°{Q) x 
C°°(r2). In particular, this pairing is well defined and separately continuous on 
A-°°{n) X A°°{n), where A-^°^{n) denotes the closure of A°{n) in 

Proposition 19. Let be a bounded domain in C" with class C°° boundary. The 
following statements are equivalent. 

1. The Bergman projection P maps the space C°°(r2) continuously into itself. 

2. The spaces yl^[°°(ri) and A°°{^1) of holomorphic functions are mutually dual 
via the extended pairing ( , ) . 



Proposition |19| is from [g6|, 110 ; the case of a strictly pseudoconve x do main is 
in and duality of spaces of harmonic functions is studied in |^l], 120|. Once 



Proposition |18| is in hand. Proposition |19| is easily proved. For example, suppose 
that the Bergman projection is known to preserve the space C°°(ri), and let t be 
a continuous linear functional on the space A~j°°(ri). Because r extends to a con- 
tinuous linear functional on the inductive limit W~^°°{^) of the ordinary Sobolev 
spaces, it is represented by pairing with a function g in the space W§°[il) of func- 
tions vanishing to infinite order at the boundary. On A^{Q), and hence on A~y°°(yL), 
pairing with g is the same as pairing with Pg since, by hypothesis, Pg S A°°{ri). 
Therefore r is indeed represented by an element of A°"{yi). 

It is nontrivial that A~^^{Q) = A~°°{yi) when Q. is pseudoconvex. Examples 
show that density properties fail dramatically in the nonpseudoconvex case ||, ||. 
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The arguments in these papers can be adapted to show that 7^ A 

for these examples. 

Theorem 20. Let Q be a bounded pseudoconvex domain in C" with class C'°° 
boundary. Then the space A°°{Q,) of holomorphic functions is dense both in A'^{il) 
and in A^'^{il) for each non-negative integer k. 

The first part is in [Q, the second in In particular, the Bergman projection 
is globally regular on a pseudoconvex domain Vl if and only if the spaces ^^°°(ri) 
and ^°°(ri) are mutually dual via the pairing ( , ). 

Here is a typical application of Proposition |l^ to the theory of the Bergman 
kernel function K{'w^z). 

Corollary 21. Let Q be a bounded domain in C" with class C°° boundary. Suppose 
that the Bergman projection maps the space C°° (SI) into itself. If S is a set of 
determinacy for holomorphic functions on il, then {K(- , z) : z Cz S} has dense 
linear span in A°°{Q). 

Indeed, global regularity of the Bergman projection P implies that K{- , z) G 
A°°(ri) for each z in fi, since K{- , z) is the projection of a smooth, radially sym- 
metric bump function (this idea originates in ||9^ ). Now if a linear functional r on 
A°°(ri) vanishes on each K{- , z) for z E S, then t{z) = on S*, whence r = 0. (Note 
that since t G ^~j°°(ri), the Bergman kernel does reproduce t, because evaluation 
at an interior point is continuous in the topology of A^°°{il.).) 

Corollary ^ is due to Bell ^ . It is the key to certain non- vanishing prop- 
erties of the Bergman kernel function that are essential in the approach to bound- 
ary regularity of holomorphic mappings developed by Bell, Ligocka, and Webster 
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